MONITOR
Results Measurement for Advisory Services
The Small and Medium Enterprise Toolkit
The Small and Medium Enterprise (SME) Toolkit was created in 2001 as part of an effort by the SME Department to find ways to reach a larger number of companies than were being reached through the traditional International Finance Corporation (IFC) strategy of working with companies on an individual basis. The theory was simple: IFC and local partners would amass information relevant to SMEs and present it on the Internet, where business people could find and use it themselves.
IFC retained Nexus Associates to provide an interim review of the Toolkit, and in 2005 the company reported that the project had accomplished its original objective and many more as well. Partners include: I IFC Facilities. The Sustainable Financial Market Facility, Latin America and Caribbean SME Facility, South Asia Enterprise Development Facility, Madagascar SME Solution Center, and the Mekong Project Development Facility view the SME Toolkit as a complement to their existing services and a means of promoting their organizations in specific markets. I Three private companies (two small IT firms and a commercial bank) use the Toolkit to attract traffic to their sites. One IT firm was spun off from IFC’s South and Eastern Europe Facility. I Non-governmental organizations have used the Toolkit to develop and enhance their presence with their clients. Examples include Business Technologies Development Center, a spin-off from PEP-Ukraine, which offers IT consulting and services to SMEs; FUNDES, a business solutions network of specialized organizations dedicated to promoting SME development in Latin America; Enterprise Development Services, a private sector initiative of the Lagos Business School created to offer consulting services and networking opportunities to SMEs in Nigeria; and the Emerging Market Private Equity Association, an organization created to help private equity and venture capital firms and institutional investors in emerging markets improve their returns and promote more investments. I Government agencies. The Small and Medium Enterprise Development Authority (SMEDA), a government agency in Pakistan—included some content and a link to the SME Toolkit on a Web site it established to reach SMEs that were not close to regional offices. The Ministry of Industry in Madagascar originally partnered with the Toolkit to launch a local content driven site now managed by the Madagascar SME Solution center.
The Toolkit at a Glance
Launched in 2001, the Toolkit was conceived as a means of providing information to SMEs in a more cost-effective manner than could be achieved through working with companies one by one. But its objectives expanded over time to include: I Building the capacity of local partners to deliver new training and consulting services. I Supporting the marketing efforts of partners. I Enabling organizations to establish a Web presence and attract visitors to their Web sites. I Establishing and reinforcing relationships between the IFC and companies in its investment portfolio. I Promoting the IFC brand. I Offering some assistance in countries where investment opportunities are limited. As of March 2005, 16 Web sites incorporated one or more components of the SME Toolkit (a number of other sites have been established since then). These include seven sites where the SME Toolkit team is fully responsible for development and maintenance of the site; five sites that are managed by local partners under terms and conditions agreed with IFC; and four sites where local partners license content from IFC that is posted—along with a broad range of other materials—on Web sites that partners have developed on their own.
Evaluation Findings
Relevance. In general, the SME Toolkit is relevant to the needs of SMEs in developing countries and IFC’s institutional objectives. The traffic on the Web sites and results of the user survey indicate that users find the content useful. Content on business planning appears most relevant, materials on accounting less so. Because the SME Toolkit is distributed via the Web, it is restricted to a subset of the population that has access to the Internet, knowledge of Microsoft Word and Excel, and the sophistication to absorb and use the materials contained in the Toolkit.
Effectiveness. The Toolkit has provided benefits to the IFC, partners and end-users. It has helped IFC promote its public image, particularly in frontier countries. It has been well received by local partners. And it has enabled some users to develop a Web presence at a relatively low cost. Many partners see the content as useful either in its own right or as a supplement to materials that they had already developed. It is reaching SMEs in developing countries and a broad range of other organizations and individuals
Who Uses the Toolkit?
A user survey, analysis of Web site traffic and interviews with partners reveal several patterns in use of the SME Toolkit. I Traffic on the SME Toolkit Web sites has been generated primarily by search engines and Web links. Overall, 41% of respondents said they found out about the Toolkit through search engines, and 33% arrived at the Toolkit sites through links from other Web sites. This reinforces the importance of optimizing the Web sites for search engines and establishing links to other sites as part of marketing campaigns. Other methods, including through newspapers or magazines, word of mouth, business training courses, and newsgroup postings were much less important, although there is substantial regional variation. In Nepal, newspaper coverage was particularly important, while for the Mongolian, Spanish, and Ukrainian sites links from other Web sites were relatively important compared to other sites. Word of mouth was relatively important for the Mongolian, Ukrainian, and Vietnamese sites. I Traffic has increased significantly. In March 2005, the number of visits to SME Toolkit sites totaled 30,000. This primarily reflects a marked increase in activity on the global and Africa sites, driven by a marketing campaign launched in August 2004. I More than half of all sessions were one minute or less, suggesting that many visitors come to the site inadvertently. But between one-fifth and one-quarter of visits lasted more than 10 minutes. I Business planning is by far the most popular topic for articles and downloads, accounting for 55 percent of the content viewed by users. I Half of all page views were requested by users in Mongolia, South Africa, the United States, and Vietnam. Besides the U.S., only three other developed countries— Canada, France, and Great Britain—were among the top 25 countries. Unresolvable IP addresses may result as US (.com) addresses so may not be reflective of U.S. based traffic. Survey respondants indicated that users tend to be well-educated males. Five times as many males use the Toolkit as women, while two-thirds of the users attended college and almost a quarter had also done post-graduate study. I Users see the Toolkit mainly as a resource library, and secondarily as a source for online training.
Many SMEs in developing countries still lack affordable access to the Internet, limiting the reach of the SME Toolkit. To mitigate this problem, the Toolkit team has created a CD-ROM version and computer-based training modules. The need for a standard Web design and content management system was debated by Nexus since some partners—FUNDES, PEP-Ukraine, NextSense, SMEDA—have developed their own Web sites using internal technical staff or external consultants. Moreover, Web design expertise is becoming easier to find throughout the world at costs that are substantially lower than in the United States or Europe. On the other hand, the Toolkit approach has probably saved money by enabling most of the partners to establish a Web presence without bearing the expense this typically entails.
Based on the Review’s survey results, 83 percent of respondents indicate that the Toolkit includes a wide range of materials that are useful to SMEs, and almost two-thirds report that the Toolkit is one of the best sites for how-to-articles and business forms. Furthermore, almost twothirds of users who had visited a Toolkit site more than once reported using it to improve their businesses. Efficiency. Given high fixed costs, the efficiency of the SME Toolkit depends on the volume of activity. As of March 31, 2005, roughly $2.0 million had been committed and $1.6 million disbursed (not including an estimated $800,000 in staff time) over the four year operational period. About 42% of total disbursements had been allocated to technology development, 17% to
partner implementation, 16% to business management training, 15% to content, 3% to general technical support, and the remaining 6% to other activities. More than $1.5 million will be invested in technical enhancements in Version 2.0 of the software, and additional expenses will be incurred in rolling out new SME Toolkit Web sites in the coming years. Much of the costs incurred to date represent start-up costs—content, Web site, content management system, and curriculum. If each partner had developed Web sites with similar content and functionality the total cost probably would have been higher. Sustainability. The SME Toolkit will depend on continued financial support from IFC and other donors. The reviewers saw little chance of spinning version 1.0 of the SME Toolkit off as a commercial venture given its emerging markets focus and use of third party content offered free of charge. No organization has yet paid for the global content, Web site design or content management system out of its own operating budget, although partners do cover their own ongoing operating costs. A number of government agencies, donors, and not-for-profit organizations have developed small-business focused Web sites, and more are likely to do so in the future. Most partners are IFC facilities, which were not designed or envisioned to become financially self-sustaining institutions; their ability to continue operations (including the further development and ongoing maintenance of the SME Toolkit) depends on support from international donors. FUNDES and SMEDA, although not IFC facilities, are similar in nature. For private companies such as NextSense, Elite Networks, and TDB, the sustainability of these Web sites depends on their ability to generate sufficient revenue or traffic to cover ongoing development and maintenance costs. Among the sources of revenue mentioned by the partners are advertising revenues from third parties and sales of principal services that come in some way through the Web site. None of the partners is currently generating significant revenues from the Toolkit directly.
Specific policies on cost sharing are needed. Partners should be required to invest a minimum amount to cover some of the start up costs of translation, initial local content, and installation and the costs of ongoing updates and maintenance of a Toolkit. To help establish this threshold, hard data must be gathered from existing partners on their actual expenditures. An ongoing mechanism to monitor partners’ investment is needed to ensure that contractual obligations are met and that the investment threshold can be adjusted periodically. Toolkit sponsors and local partners should regularly add content on areas of interest to SMEs in developing countries. A multitude of content providers around the world might be interested in licensing materials for non-commercial purposes as part of good citizenship or public relations campaigns. Those responsible for managing content should make sure that materials, including more interactive materials for training purposes, are at a level that less sophisticated users can understand and use. In the near term, more content is needed related to business planning, which currently is the most popular subject on current sites. For the longer term, a systematic process is needed for collecting feedback from partners and users concerning their content requirements. As part of this process, managers should assess whether low usage of some materials is due to limited interest, lack of relevance, or poor quality. Assuming that a registration system is adopted, user feedback should be based on regular, randomized surveys. (Version 2 of the SME Toolkit has been designed to enable full-implementation partners to update their Web sites more easily and share content with other SME Toolkit partners.) Information providers should develop multiple distribution channels for the SME Toolkit content. Most partners involved with the SME Toolkit indicated that access to the Internet in their markets is limited, slow and costly. A possible solution would be to distribute content on CD-ROMs on a much broader scale. To date, the Toolkit team and local partners have made only limited use of CD-ROMs, distributing approximately 3,000 mainly for promotional purposes. Organizations that are working with SMEs, particularly those that receive funding or technical assistance from the World Bank, would be obvious candidates for such distribution. As of March 2005, all the global content of the SME Toolkit was available in English, French, Mongolian, Spanish, and portions had been translated into other languages. Planned enhancements for Version 2 of the SME Toolkit will make it much easier for the Toolkit team and partners to publish the SME Toolkit on CD-ROM. Web sites and CD-ROMs also should be considered as channels to distribute training manuals to BDS providers, enabling more organizations to learn how to use the SME Toolkit in training courses.
Lessons Learned
The SME Toolkit is a useful product that can serve both institutional and development objectives. Sponsoring organizations can use the Toolkit to establish a Web presence, advertise their services, post information relevant to SMEs, and otherwise support their partners and clients. The ideal licensees are organizations that are committed to significant localization and ongoing maintenance of the Web site. Whether partners undertake full implementation or just use Toolkit content on their own web sites, they should be responsible for adding content that is valuable to SMEs in their markets and not widely available on other sites. Agreements should specify the nature of the content that the local partner will provide initially, plans for adding additional content in the future, and a schedule for implementation.
Marketing is key to the success of products like the SME Toolkit. The SME Toolkit team has successfully initiated efforts to increase traffic on certain Web sites, but as of March 2005, traffic on many sites was low. Since most users are likely to reach the Web sites through search engines or links from other Web sites, the design and text of all sites should be optimized based on the algorithm that Google uses for ranking. Moreover, Web sites should be updated on a regular basis with information of greatest interest to companies in the target market. Content should be added not only to benefit users, but also to increase rankings by search engines. Other marketing approaches— direct mail, broadcast media, and press coverage—should be the responsibility of local partners and be tailored to the particular market characteristics in each country. Performance metrics for the SME Toolkit need to be aligned with objectives. To ensure that the products meet the needs of managers, donors, partner organizations, and SMEs, annual targets should be set: the number of sites by business model (co-brand, full implementation, contentonly), number of sites by partner type (IFC facilities, not-for profit organizations, for-profit private enterprises), unique visitors, frequency of visits, number of sessions, percentage of sessions from developing countries, average duration of sessions, percentage of sessions lasting more than ten minutes, average depth of session (that is, number of page views), percentage of sessions with more than ten page views, percentage of users reporting that information has been useful, percentage of users reporting that information is not readily accessible elsewhere, total cost per session hour, and total cost per participanttraining hour. Results should be reported annually.
Long-term strategic plans and annual operating plans should set explicit goals and objectives. A strategic plan should define the product and target market precisely; provide a clear statement of guiding principles; define major policies; describe the management structure and staffing (including consultants); define how important and recurrent activities will be carried out; specify needed equipment and software; define the investment needed to develop, deploy, and maintain the Web site; explain how costs will be covered; and delineate the performance measures and standards that will be used to gauge whether the organization is achieving its objectives. The strategic plan should also delineate potential exit strategies in the event that donors are no longer willing to support a project. Annual operating plans should detail the specific activities that will be carried out. Since there is no price mechanism to mediate demand, managers should ensure that partners have explicit plans for using additional features and functionality before making an investment. The strategy also should clearly articulate the role of managers in training trainers, developing curricula for classroom-based management training, and delivering courses directly to end users. And it should describe how such training efforts can be replicated to maximize the return on training investments.
Monitor Monitor shares key findings from in depth reviews of advisory services programs and projects conducted by external evaluators.These reviews address the relevance, efficiency, effectiveness and sustainability of the Advisory services programs. Results Measurement Unit of the SME department Resultsmeasurement@ifc.org